Papers with self-attention mechanism complexity
On the Locality of Attention in Direct Speech Translation (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent advances in NLP have created problems with the complexity of the self-attention layer. |
| Approach: | They propose to substitute standard self-attention with a local efficient one to avoid the computation of attention weights. |
| Outcome: | The proposed model matches the baseline performance and improves efficiency by skipping the computation of weights that standard attention discards. |